{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"13_Recurrent_Neural_Networks","provenance":[{"file_id":"https://github.com/GokuMohandas/MadeWithML/blob/main/notebooks/13_Recurrent_Neural_Networks.ipynb","timestamp":1608297924248},{"file_id":"https://github.com/GokuMohandas/MadeWithML/blob/main/notebooks/13_Recurrent_Neural_Networks.ipynb","timestamp":1584546246533},{"file_id":"https://github.com/GokuMohandas/MadeWithML/blob/main/notebooks/13_Recurrent_Neural_Networks.ipynb","timestamp":1583255247211}],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"mgZTKjbZ6kfz"},"source":["<div align=\"center\">\n","<h1><img width=\"30\" src=\"../res/maiwei.png\">&nbsp;<a href=\"https://github.com/Charmve/computer-vision-in-action\">Coumpter Vision in Action</a></h1>\n","Applied ML · MLOps · Production\n","<br>\n","Join 20K+ developers in learning how to responsibly <a href=\"https://madewithml.com/about/\">deliver value</a> with ML.\n","</div>\n","\n","<br>\n","\n","<div align=\"center\">\n","    <a target=\"_blank\" href=\"https://newsletter.madewithml.com\"><img src=\"https://img.shields.io/badge/Subscribe-20K-brightgreen\"></a>&nbsp;\n","    <a target=\"_blank\" href=\"https://github.com/GokuMohandas/MadeWithML\"><img src=\"https://img.shields.io/github/stars/GokuMohandas/MadeWithML.svg?style=social&label=Star\"></a>&nbsp;\n","    <a target=\"_blank\" href=\"https://www.linkedin.com/in/goku\"><img src=\"https://img.shields.io/badge/style--5eba00.svg?label=LinkedIn&logo=linkedin&style=social\"></a>&nbsp;\n","    <a target=\"_blank\" href=\"https://twitter.com/GokuMohandas\"><img src=\"https://img.shields.io/twitter/follow/GokuMohandas.svg?label=Follow&style=social\"></a>\n","    <p>🔥&nbsp; Among the <a href=\"https://github.com/topics/deep-learning\" target=\"_blank\">top ML</a> repositories on GitHub</p>\n","</div>\n","\n","<br>\n","<hr>"]},{"cell_type":"markdown","metadata":{"id":"eTdCMVl9YAXw"},"source":["# Recurrent Neural Networks (RNN)\n","\n","In this lesson we will learn how to process sequential data (sentences, time-series, etc.) with recurrent neural networks (RNNs)."]},{"cell_type":"markdown","metadata":{"id":"xuabAj4PYj57"},"source":["<div align=\"left\">\n","<a target=\"_blank\" href=\"https://madewithml.com/courses/ml-foundations/recurrent-neural-networks/\"><img src=\"https://img.shields.io/badge/📖 Read-blog post-9cf\"></a>&nbsp;\n","<a href=\"https://github.com/Charmve/computer-vision-in-action/blob/main/notebooks/chapter04_recurrent-neural-networks/Recurrent_Neural_Networks.ipynb\" role=\"button\"><img src=\"https://img.shields.io/static/v1?label=&amp;message=View%20On%20GitHub&amp;color=586069&amp;logo=github&amp;labelColor=2f363d\"></a>&nbsp;\n","<a href=\"https://colab.research.google.com/github/Charmve/computer-vision-in-action/blob/main/notebooks/chapter04_recurrent-neural-networks/Recurrent_Neural_Networks.ipynb\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n","</div>"]},{"cell_type":"markdown","metadata":{"id":"uXuhm14WaCxy"},"source":["So far we've processed inputs as whole (ex. applying filters across the entire input to extract features) but we can also process our inputs sequentially. For example we can think of each token in our text as an event in time (timestep). We can process each timestep, one at a time, and predict the class after the last timestep (token) has been processed. This is very powerful because the model now has a meaningful way to account for the sequential order of tokens in our sequence and predict accordingly. "]},{"cell_type":"markdown","metadata":{"id":"m5JJY0tl8StF"},"source":["# Overview"]},{"cell_type":"markdown","metadata":{"id":"34Yf1pdURtLn"},"source":["<div align=\"left\">\n","<img src=\"https://raw.githubusercontent.com/GokuMohandas/MadeWithML/main/images/basics/rnn/vanilla.png\" width=\"500\">\n","</div>\n","\n","RNN forward pass for a single time step $X_t$:\n","\n","$h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h)$\n","\n","*where*:\n","* $W_{hh}$ = hidden units weights| $\\in \\mathbb{R}^{HXH}$ ($H$ is the hidden dim)\n","* $h_{t-1}$ = previous timestep's hidden state $\\in \\mathbb{R}^{NXH}$\n","* $W_{xh}$ = input weights| $\\in \\mathbb{R}^{EXH}$\n","* $X_t$ = input at time step t | $\\in \\mathbb{R}^{NXE}$ ($N$ is the batch size, $E$ is the embedding dim)\n","* $b_h$ = hidden units bias $\\in \\mathbb{R}^{HX1}$\n","* $h_t$ = output from RNN for timestep $t$"]},{"cell_type":"markdown","metadata":{"id":"JqxyljU18hvt"},"source":["* **Objective:**  Process sequential data by accounting for the currend input and also what has been learned from previous inputs.\n","* **Advantages:** \n","    * Account for order and previous inputs in a meaningful way.\n","    * Conditioned generation for generating sequences.\n","* **Disadvantages:** \n","    * Each time step's prediction depends on the previous prediction so it's difficult to parallelize RNN operations. \n","    * Processing long sequences can yield memory and computation issues.\n","    * Interpretability is difficult but there are few [techniques](https://arxiv.org/abs/1506.02078) that use the activations from RNNs to see what parts of the inputs are processed. \n","* **Miscellaneous:** \n","    * Architectural tweaks to make RNNs faster and interpretable is an ongoing area of research."]},{"cell_type":"markdown","metadata":{"id":"V88LZeTqDePU"},"source":["# Set up"]},{"cell_type":"code","metadata":{"id":"DlRv80v5DfdP"},"source":["import numpy as np\n","import pandas as pd\n","import random\n","import torch\n","import torch.nn as nn"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"NoNhL0wTDgLW"},"source":["SEED = 1234"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"gwVLi0UPDgOd"},"source":["def set_seeds(seed=1234):\n","    \"\"\"Set seeds for reproducibility.\"\"\"\n","    np.random.seed(seed)\n","    random.seed(seed)\n","    torch.manual_seed(seed)\n","    torch.cuda.manual_seed(seed)\n","    torch.cuda.manual_seed_all(seed) # multi-GPU# Set seeds for reproducibility\n","set_seeds(seed=SEED)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"HK5Onfz9DiPO"},"source":["# Set seeds for reproducibility\n","set_seeds(seed=SEED)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"4T1rZDhsDiRt","executionInfo":{"status":"ok","timestamp":1608336585403,"user_tz":420,"elapsed":3999,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"915410d9-f37f-4d86-bd6d-2cbf95352658"},"source":["# Set device\n","cuda = True\n","device = torch.device('cuda' if (\n","    torch.cuda.is_available() and cuda) else 'cpu')\n","torch.set_default_tensor_type('torch.FloatTensor')\n","if device.type == 'cuda':\n","    torch.set_default_tensor_type('torch.cuda.FloatTensor')\n","print (device)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["cuda\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"c69z9wpJ56nE"},"source":["## Load data"]},{"cell_type":"markdown","metadata":{"id":"2V_nEp5G58M0"},"source":["We will download the [AG News dataset](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html), which consists of 120K text samples from 4 unique classes (`Business`, `Sci/Tech`, `Sports`, `World`)"]},{"cell_type":"code","metadata":{"id":"y3qKSoEe57na"},"source":["import numpy as np\n","import pandas as pd\n","import re\n","import urllib"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"BqaHCtToXWAb","colab":{"base_uri":"https://localhost:8080/","height":204},"executionInfo":{"status":"ok","timestamp":1608336585908,"user_tz":420,"elapsed":4479,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"533545e7-f422-4e0c-9e35-54de9481fde1"},"source":["# Load data\n","url = \"https://raw.githubusercontent.com/GokuMohandas/MadeWithML/main/datasets/news.csv\"\n","df = pd.read_csv(url, header=0) # load\n","df = df.sample(frac=1).reset_index(drop=True) # shuffle\n","df.head()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>title</th>\n","      <th>category</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Sharon Accepts Plan to Reduce Gaza Army Operat...</td>\n","      <td>World</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Internet Key Battleground in Wildlife Crime Fight</td>\n","      <td>Sci/Tech</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>July Durable Good Orders Rise 1.7 Percent</td>\n","      <td>Business</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Growing Signs of a Slowing on Wall Street</td>\n","      <td>Business</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>The New Faces of Reality TV</td>\n","      <td>World</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                               title  category\n","0  Sharon Accepts Plan to Reduce Gaza Army Operat...     World\n","1  Internet Key Battleground in Wildlife Crime Fight  Sci/Tech\n","2          July Durable Good Orders Rise 1.7 Percent  Business\n","3          Growing Signs of a Slowing on Wall Street  Business\n","4                        The New Faces of Reality TV     World"]},"metadata":{"tags":[]},"execution_count":7}]},{"cell_type":"markdown","metadata":{"id":"HR_lLiVBDxW1"},"source":["## Preprocessing"]},{"cell_type":"markdown","metadata":{"id":"u00AZ0O8DxaY"},"source":["We're going to clean up our input data first by doing operations such as lower text, removing stop (filler) words, filters using regular expressions, etc."]},{"cell_type":"code","metadata":{"id":"Jq6a11YsDvEU"},"source":["import nltk\n","from nltk.corpus import stopwords\n","from nltk.stem import PorterStemmer\n","import re"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"x5IpVaKqDvG1","executionInfo":{"status":"ok","timestamp":1608336587020,"user_tz":420,"elapsed":5564,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"19513aa1-3bfd-4770-956c-626951a6678c"},"source":["nltk.download('stopwords')\n","STOPWORDS = stopwords.words('english')\n","print (STOPWORDS[:5])\n","porter = PorterStemmer()"],"execution_count":null,"outputs":[{"output_type":"stream","text":["[nltk_data] Downloading package stopwords to /root/nltk_data...\n","[nltk_data]   Unzipping corpora/stopwords.zip.\n","['i', 'me', 'my', 'myself', 'we']\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"1n4w3KLQDvJk"},"source":["def preprocess(text, stopwords=STOPWORDS):\n","    \"\"\"Conditional preprocessing on our text unique to our task.\"\"\"\n","    # Lower\n","    text = text.lower()\n","\n","    # Remove stopwords\n","    pattern = re.compile(r'\\b(' + r'|'.join(stopwords) + r')\\b\\s*')\n","    text = pattern.sub('', text)\n","\n","    # Remove words in paranthesis\n","    text = re.sub(r'\\([^)]*\\)', '', text)\n","\n","    # Spacing and filters\n","    text = re.sub(r\"([-;;.,!?<=>])\", r\" \\1 \", text)\n","    text = re.sub('[^A-Za-z0-9]+', ' ', text) # remove non alphanumeric chars\n","    text = re.sub(' +', ' ', text)  # remove multiple spaces\n","    text = text.strip()\n","\n","    return text"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":35},"id":"U9JmpiCmDvMz","executionInfo":{"status":"ok","timestamp":1608336587022,"user_tz":420,"elapsed":5542,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"e60b782d-32d7-4cfb-cff2-9404e078e439"},"source":["# Sample\n","text = \"Great week for the NYSE!\"\n","preprocess(text=text)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"},"text/plain":["'great week nyse'"]},"metadata":{"tags":[]},"execution_count":11}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"8RZ45-N9DvQD","executionInfo":{"status":"ok","timestamp":1608336589840,"user_tz":420,"elapsed":8344,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"f756d230-fe95-4532-8497-d66759534e46"},"source":["# Apply to dataframe\n","preprocessed_df = df.copy()\n","preprocessed_df.title = preprocessed_df.title.apply(preprocess)\n","print (f\"{df.title.values[0]}\\n\\n{preprocessed_df.title.values[0]}\")"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Sharon Accepts Plan to Reduce Gaza Army Operation, Haaretz Says\n","\n","sharon accepts plan reduce gaza army operation haaretz says\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"hxo6RKCQ71dl"},"source":["## Split data"]},{"cell_type":"code","metadata":{"id":"eS6kCcfY6IHE"},"source":["import collections\n","from sklearn.model_selection import train_test_split"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"4qkWnO3uYPnk"},"source":["TRAIN_SIZE = 0.7\n","VAL_SIZE = 0.15\n","TEST_SIZE = 0.15"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"0gbh2TLBXVDu"},"source":["def train_val_test_split(X, y, train_size):\n","    \"\"\"Split dataset into data splits.\"\"\"\n","    X_train, X_, y_train, y_ = train_test_split(X, y, train_size=TRAIN_SIZE, stratify=y)\n","    X_val, X_test, y_val, y_test = train_test_split(X_, y_, train_size=0.5, stratify=y_)\n","    return X_train, X_val, X_test, y_train, y_val, y_test"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"kqiQd2j_76gP"},"source":["# Data\n","X = preprocessed_df[\"title\"].values\n","y = preprocessed_df[\"category\"].values"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"AwdPM136EBMA","executionInfo":{"status":"ok","timestamp":1608336589843,"user_tz":420,"elapsed":8299,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"05a3e7bc-7edb-4363-c990-7c5d94f7e037"},"source":["# Create data splits\n","X_train, X_val, X_test, y_train, y_val, y_test = train_val_test_split(\n","    X=X, y=y, train_size=TRAIN_SIZE)\n","print (f\"X_train: {X_train.shape}, y_train: {y_train.shape}\")\n","print (f\"X_val: {X_val.shape}, y_val: {y_val.shape}\")\n","print (f\"X_test: {X_test.shape}, y_test: {y_test.shape}\")\n","print (f\"Sample point: {X_train[0]} → {y_train[0]}\")"],"execution_count":null,"outputs":[{"output_type":"stream","text":["X_train: (84000,), y_train: (84000,)\n","X_val: (18000,), y_val: (18000,)\n","X_test: (18000,), y_test: (18000,)\n","Sample point: china battles north korea nuclear talks → World\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"kUAn5IOjEF8P"},"source":["## LabelEncoder"]},{"cell_type":"markdown","metadata":{"id":"fNTjndI0EKzm"},"source":["Next we'll define a `LabelEncoder` to encode our text labels into unique indices"]},{"cell_type":"code","metadata":{"id":"OgLNA5yTELB1"},"source":["import itertools"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"GeoPR77QELJT"},"source":["class LabelEncoder(object):\n","    \"\"\"Label encoder for tag labels.\"\"\"\n","    def __init__(self, class_to_index={}):\n","        self.class_to_index = class_to_index\n","        self.index_to_class = {v: k for k, v in self.class_to_index.items()}\n","        self.classes = list(self.class_to_index.keys())\n","\n","    def __len__(self):\n","        return len(self.class_to_index)\n","\n","    def __str__(self):\n","        return f\"<LabelEncoder(num_classes={len(self)})>\"\n","\n","    def fit(self, y):\n","        classes = np.unique(y)\n","        for i, class_ in enumerate(classes):\n","            self.class_to_index[class_] = i\n","        self.index_to_class = {v: k for k, v in self.class_to_index.items()}\n","        self.classes = list(self.class_to_index.keys())\n","        return self\n","\n","    def encode(self, y):\n","        encoded = np.zeros((len(y)), dtype=int)\n","        for i, item in enumerate(y):\n","            encoded[i] = self.class_to_index[item]\n","        return encoded\n","\n","    def decode(self, y):\n","        classes = []\n","        for i, item in enumerate(y):\n","            classes.append(self.index_to_class[item])\n","        return classes\n","\n","    def save(self, fp):\n","        with open(fp, 'w') as fp:\n","            contents = {'class_to_index': self.class_to_index}\n","            json.dump(contents, fp, indent=4, sort_keys=False)\n","\n","    @classmethod\n","    def load(cls, fp):\n","        with open(fp, 'r') as fp:\n","            kwargs = json.load(fp=fp)\n","        return cls(**kwargs)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"0MtTxMB5ELL4","executionInfo":{"status":"ok","timestamp":1608336589845,"user_tz":420,"elapsed":8268,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"f8cff1ae-27f6-4a43-d0ef-f77580a06aef"},"source":["# Encode\n","label_encoder = LabelEncoder()\n","label_encoder.fit(y_train)\n","NUM_CLASSES = len(label_encoder)\n","label_encoder.class_to_index"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["{'Business': 0, 'Sci/Tech': 1, 'Sports': 2, 'World': 3}"]},"metadata":{"tags":[]},"execution_count":20}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"0IzPFnoDELSr","executionInfo":{"status":"ok","timestamp":1608336589846,"user_tz":420,"elapsed":8252,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"9b7d53e7-c407-419a-caba-8cfc3aa38f50"},"source":["# Convert labels to tokens\n","print (f\"y_train[0]: {y_train[0]}\")\n","y_train = label_encoder.encode(y_train)\n","y_val = label_encoder.encode(y_val)\n","y_test = label_encoder.encode(y_test)\n","print (f\"y_train[0]: {y_train[0]}\")"],"execution_count":null,"outputs":[{"output_type":"stream","text":["y_train[0]: World\n","y_train[0]: 3\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"xjESU2TOEQz_","executionInfo":{"status":"ok","timestamp":1608336589846,"user_tz":420,"elapsed":8236,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"ca868130-129b-40f6-9c77-b677d0ff3b87"},"source":["# Class weights\n","counts = np.bincount(y_train)\n","class_weights = {i: 1.0/count for i, count in enumerate(counts)}\n","print (f\"counts: {counts}\\nweights: {class_weights}\")"],"execution_count":null,"outputs":[{"output_type":"stream","text":["counts: [21000 21000 21000 21000]\n","weights: {0: 4.761904761904762e-05, 1: 4.761904761904762e-05, 2: 4.761904761904762e-05, 3: 4.761904761904762e-05}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"dIfmW7vJ8Jx1"},"source":["## Tokenizer"]},{"cell_type":"markdown","metadata":{"id":"IYREXERdETdQ"},"source":["We'll define a `Tokenizer` to convert our text input data into token indices."]},{"cell_type":"code","metadata":{"id":"0V6v1yeVGxNg"},"source":["import json\n","from collections import Counter\n","from more_itertools import take"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"ChhTN8WDY4Z5"},"source":["class Tokenizer(object):\n","    def __init__(self, char_level, num_tokens=None, \n","                 pad_token='<PAD>', oov_token='<UNK>',\n","                 token_to_index=None):\n","        self.char_level = char_level\n","        self.separator = '' if self.char_level else ' '\n","        if num_tokens: num_tokens -= 2 # pad + unk tokens\n","        self.num_tokens = num_tokens\n","        self.pad_token = pad_token\n","        self.oov_token = oov_token\n","        if not token_to_index:\n","            token_to_index = {pad_token: 0, oov_token: 1}\n","        self.token_to_index = token_to_index\n","        self.index_to_token = {v: k for k, v in self.token_to_index.items()}\n","\n","    def __len__(self):\n","        return len(self.token_to_index)\n","\n","    def __str__(self):\n","        return f\"<Tokenizer(num_tokens={len(self)})>\"\n","\n","    def fit_on_texts(self, texts):\n","        if not self.char_level:\n","            texts = [text.split(\" \") for text in texts]\n","        all_tokens = [token for text in texts for token in text]\n","        counts = Counter(all_tokens).most_common(self.num_tokens)\n","        self.min_token_freq = counts[-1][1]\n","        for token, count in counts:\n","            index = len(self)\n","            self.token_to_index[token] = index\n","            self.index_to_token[index] = token\n","        return self\n","\n","    def texts_to_sequences(self, texts):\n","        sequences = []\n","        for text in texts:\n","            if not self.char_level:\n","                text = text.split(' ')\n","            sequence = []\n","            for token in text:\n","                sequence.append(self.token_to_index.get(\n","                    token, self.token_to_index[self.oov_token]))\n","            sequences.append(np.asarray(sequence))\n","        return sequences\n","\n","    def sequences_to_texts(self, sequences):\n","        texts = []\n","        for sequence in sequences:\n","            text = []\n","            for index in sequence:\n","                text.append(self.index_to_token.get(index, self.oov_token))\n","            texts.append(self.separator.join([token for token in text]))\n","        return texts\n","\n","    def save(self, fp):\n","        with open(fp, 'w') as fp:\n","            contents = {\n","                'char_level': self.char_level,\n","                'oov_token': self.oov_token,\n","                'token_to_index': self.token_to_index\n","            }\n","            json.dump(contents, fp, indent=4, sort_keys=False)\n","\n","    @classmethod\n","    def load(cls, fp):\n","        with open(fp, 'r') as fp:\n","            kwargs = json.load(fp=fp)\n","        return cls(**kwargs)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"ylpHSbcZYsXa","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608336589848,"user_tz":420,"elapsed":8205,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"5cb753ce-71ca-416c-f15b-9e7eaf9f42b1"},"source":["# Tokenize\n","tokenizer = Tokenizer(char_level=False, num_tokens=5000)\n","tokenizer.fit_on_texts(texts=X_train)\n","VOCAB_SIZE = len(tokenizer)\n","print (tokenizer)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["<Tokenizer(num_tokens=5000)>\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"bVgBSnFTfE5D","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608336589848,"user_tz":420,"elapsed":8190,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"1e531cd2-1644-44a3-c58c-45850d1ede67"},"source":["# Sample of tokens\n","print (take(5, tokenizer.token_to_index.items()))\n","print (f\"least freq token's freq: {tokenizer.min_token_freq}\") # use this to adjust num_tokens"],"execution_count":null,"outputs":[{"output_type":"stream","text":["[('<PAD>', 0), ('<UNK>', 1), ('39', 2), ('b', 3), ('gt', 4)]\n","least freq token's freq: 14\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"WcscM_vL8KvP","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608336590523,"user_tz":420,"elapsed":8849,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"e11d9f5f-3058-44d0-eb39-5667d8b51762"},"source":["# Convert texts to sequences of indices\n","X_train = tokenizer.texts_to_sequences(X_train)\n","X_val = tokenizer.texts_to_sequences(X_val)\n","X_test = tokenizer.texts_to_sequences(X_test)\n","preprocessed_text = tokenizer.sequences_to_texts([X_train[0]])[0]\n","print (\"Text to indices:\\n\"\n","    f\"  (preprocessed) → {preprocessed_text}\\n\"\n","    f\"  (tokenized) → {X_train[0]}\")"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Text to indices:\n","  (preprocessed) → china battles north korea nuclear talks\n","  (tokenized) → [  16 1491  285  142  114   24]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"9o91b6kKHjN4"},"source":["## Padding"]},{"cell_type":"markdown","metadata":{"id":"bD6CuPxOHlsw"},"source":["We'll need to do 2D padding to our tokenized text."]},{"cell_type":"code","metadata":{"id":"dcNEooUHHkwn"},"source":["def pad_sequences(sequences, max_seq_len=0):\n","    \"\"\"Pad sequences to max length in sequence.\"\"\"\n","    max_seq_len = max(max_seq_len, max(len(sequence) for sequence in sequences))\n","    padded_sequences = np.zeros((len(sequences), max_seq_len))\n","    for i, sequence in enumerate(sequences):\n","        padded_sequences[i][:len(sequence)] = sequence\n","    return padded_sequences"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sfWHamHZHtT-","executionInfo":{"status":"ok","timestamp":1608336590525,"user_tz":420,"elapsed":8827,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"43ef805c-6c70-4a2a-a46d-e6f0f7c28dee"},"source":["# 2D sequences\n","padded = pad_sequences(X_train[0:3])\n","print (padded.shape)\n","print (padded)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["(3, 6)\n","[[1.600e+01 1.491e+03 2.850e+02 1.420e+02 1.140e+02 2.400e+01]\n"," [1.445e+03 2.300e+01 6.560e+02 2.197e+03 1.000e+00 0.000e+00]\n"," [1.200e+02 1.400e+01 1.955e+03 1.005e+03 1.529e+03 4.014e+03]]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"eoWQk0hO9bK2"},"source":["## Datasets"]},{"cell_type":"markdown","metadata":{"id":"dSPbGwOkHvfj"},"source":["We're going to create Datasets and DataLoaders to be able to efficiently create batches with our data splits."]},{"cell_type":"code","metadata":{"id":"GVxnbzgW8X1V"},"source":["FILTER_SIZES = list(range(1, 4)) # uni, bi and tri grams"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"aSdfc0-cb9fc"},"source":["class Dataset(torch.utils.data.Dataset):\n","    def __init__(self, X, y, max_filter_size):\n","        self.X = X\n","        self.y = y\n","        self.max_filter_size = max_filter_size\n","\n","    def __len__(self):\n","        return len(self.y)\n","\n","    def __str__(self):\n","        return f\"<Dataset(N={len(self)})>\"\n","\n","    def __getitem__(self, index):\n","        X = self.X[index]\n","        y = self.y[index]\n","        return [X, len(X), y]\n","\n","    def collate_fn(self, batch):\n","        \"\"\"Processing on a batch.\"\"\"\n","        # Get inputs\n","        batch = np.array(batch, dtype=object)\n","        X = batch[:, 0]\n","        seq_lens = batch[:, 1]\n","        y = np.stack(batch[:, 2], axis=0)\n","\n","        # Pad inputs\n","        X = pad_sequences(sequences=X)\n","\n","        # Cast\n","        X = torch.LongTensor(X.astype(np.int32))\n","        seq_lens = torch.LongTensor(seq_lens.astype(np.int32))\n","        y = torch.LongTensor(y.astype(np.int32))\n","\n","        return X, seq_lens, y\n","\n","    def create_dataloader(self, batch_size, shuffle=False, drop_last=False):\n","        return torch.utils.data.DataLoader(\n","            dataset=self, batch_size=batch_size, collate_fn=self.collate_fn,\n","            shuffle=shuffle, drop_last=drop_last, pin_memory=True)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"KYZtzbWMIXY7","executionInfo":{"status":"ok","timestamp":1608336590526,"user_tz":420,"elapsed":8794,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"06cb6dad-d344-4e94-b3b9-150d4d418c0e"},"source":["# Create datasets\n","max_filter_size = max(FILTER_SIZES)\n","train_dataset = Dataset(X=X_train, y=y_train, max_filter_size=max_filter_size)\n","val_dataset = Dataset(X=X_val, y=y_val, max_filter_size=max_filter_size)\n","test_dataset = Dataset(X=X_test, y=y_test, max_filter_size=max_filter_size)\n","print (\"Datasets:\\n\"\n","    f\"  Train dataset:{train_dataset.__str__()}\\n\"\n","    f\"  Val dataset: {val_dataset.__str__()}\\n\"\n","    f\"  Test dataset: {test_dataset.__str__()}\\n\"\n","    \"Sample point:\\n\"\n","    f\"  X: {train_dataset[0][0]}\\n\"\n","    f\"  seq_len: {train_dataset[0][1]}\\n\"\n","    f\"  y: {train_dataset[0][2]}\")"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Datasets:\n","  Train dataset:<Dataset(N=84000)>\n","  Val dataset: <Dataset(N=18000)>\n","  Test dataset: <Dataset(N=18000)>\n","Sample point:\n","  X: [  16 1491  285  142  114   24]\n","  seq_len: 6\n","  y: 3\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"1w6wVKJe9fxk","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608336601044,"user_tz":420,"elapsed":19296,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"4d6e148a-7c18-4552-c6bd-7e27c6b9b6c6"},"source":["# Create dataloaders\n","batch_size = 64\n","train_dataloader = train_dataset.create_dataloader(\n","    batch_size=batch_size)\n","val_dataloader = val_dataset.create_dataloader(\n","    batch_size=batch_size)\n","test_dataloader = test_dataset.create_dataloader(\n","    batch_size=batch_size)\n","batch_X, batch_seq_lens, batch_y = next(iter(train_dataloader))\n","print (\"Sample batch:\\n\"\n","    f\"  X: {list(batch_X.size())}\\n\"\n","    f\"  seq_lens: {list(batch_seq_lens.size())}\\n\"\n","    f\"  y: {list(batch_y.size())}\\n\"\n","    \"Sample point:\\n\"\n","    f\"  X: {batch_X[0]}\\n\"\n","    f\" seq_len: {batch_seq_lens[0]}\\n\"\n","    f\"  y: {batch_y[0]}\")"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Sample batch:\n","  X: [64, 14]\n","  seq_lens: [64]\n","  y: [64]\n","Sample point:\n","  X: tensor([  16, 1491,  285,  142,  114,   24,    0,    0,    0,    0,    0,    0,\n","           0,    0], device='cpu')\n"," seq_len: 6\n","  y: 3\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"dB-g9F1TJOPx"},"source":["## Trainer"]},{"cell_type":"markdown","metadata":{"id":"dL5FcX0hJR37"},"source":["Let's create the `Trainer` class that we'll use to facilitate training for our experiments."]},{"cell_type":"code","metadata":{"id":"k5KWF-DXJN2a"},"source":["class Trainer(object):\n","    def __init__(self, model, device, loss_fn=None, optimizer=None, scheduler=None):\n","\n","        # Set params\n","        self.model = model\n","        self.device = device\n","        self.loss_fn = loss_fn\n","        self.optimizer = optimizer\n","        self.scheduler = scheduler\n","\n","    def train_step(self, dataloader):\n","        \"\"\"Train step.\"\"\"\n","        # Set model to train mode\n","        self.model.train()\n","        loss = 0.0\n","\n","        # Iterate over train batches\n","        for i, batch in enumerate(dataloader):\n","\n","            # Step\n","            batch = [item.to(self.device) for item in batch]  # Set device\n","            inputs, targets = batch[:-1], batch[-1]\n","            self.optimizer.zero_grad()  # Reset gradients\n","            z = self.model(inputs)  # Forward pass\n","            J = self.loss_fn(z, targets)  # Define loss\n","            J.backward()  # Backward pass\n","            self.optimizer.step()  # Update weights\n","\n","            # Cumulative Metrics\n","            loss += (J.detach().item() - loss) / (i + 1)\n","\n","        return loss\n","\n","    def eval_step(self, dataloader):\n","        \"\"\"Validation or test step.\"\"\"\n","        # Set model to eval mode\n","        self.model.eval()\n","        loss = 0.0\n","        y_trues, y_probs = [], []\n","\n","        # Iterate over val batches\n","        with torch.no_grad():\n","            for i, batch in enumerate(dataloader):\n","\n","                # Step\n","                batch = [item.to(self.device) for item in batch]  # Set device\n","                inputs, y_true = batch[:-1], batch[-1]\n","                z = self.model(inputs)  # Forward pass\n","                J = self.loss_fn(z, y_true).item()\n","\n","                # Cumulative Metrics\n","                loss += (J - loss) / (i + 1)\n","\n","                # Store outputs\n","                y_prob = torch.sigmoid(z).cpu().numpy()\n","                y_probs.extend(y_prob)\n","                y_trues.extend(y_true.cpu().numpy())\n","\n","        return loss, np.vstack(y_trues), np.vstack(y_probs)\n","\n","    def predict_step(self, dataloader):\n","        \"\"\"Prediction step.\"\"\"\n","        # Set model to eval mode\n","        self.model.eval()\n","        y_probs = []\n","\n","        # Iterate over val batches\n","        with torch.no_grad():\n","            for i, batch in enumerate(dataloader):\n","\n","                # Forward pass w/ inputs\n","                inputs, targets = batch[:-1], batch[-1]\n","                y_prob = self.model(inputs, apply_softmax=True)\n","\n","                # Store outputs\n","                y_probs.extend(y_prob)\n","\n","        return np.vstack(y_probs)\n","    \n","    def train(self, num_epochs, patience, train_dataloader, val_dataloader):\n","        best_val_loss = np.inf\n","        for epoch in range(num_epochs):\n","            # Steps\n","            train_loss = self.train_step(dataloader=train_dataloader)\n","            val_loss, _, _ = self.eval_step(dataloader=val_dataloader)\n","            self.scheduler.step(val_loss)\n","\n","            # Early stopping\n","            if val_loss < best_val_loss:\n","                best_val_loss = val_loss\n","                best_model = self.model\n","                _patience = patience  # reset _patience\n","            else:\n","                _patience -= 1\n","            if not _patience:  # 0\n","                print(\"Stopping early!\")\n","                break\n","\n","            # Logging\n","            print(\n","                f\"Epoch: {epoch+1} | \"\n","                f\"train_loss: {train_loss:.5f}, \"\n","                f\"val_loss: {val_loss:.5f}, \"\n","                f\"lr: {self.optimizer.param_groups[0]['lr']:.2E}, \"\n","                f\"_patience: {_patience}\"\n","            )\n","        return best_model"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"bUMibnFtuq_i"},"source":["# Vanilla RNN"]},{"cell_type":"markdown","metadata":{"id":"EwyHfs4JvC7F"},"source":["Inputs to RNNs are sequential like text or time-series."]},{"cell_type":"code","metadata":{"id":"d3e6DbkwhXow"},"source":["BATCH_SIZE = 64\n","EMBEDDING_DIM = 100"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"JOs64ORxvBei","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608336601046,"user_tz":420,"elapsed":19267,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"bb38d712-4fea-4f78-e9a9-78a08b4b2192"},"source":["# Input\n","sequence_size = 8 # words per input\n","x = torch.rand((BATCH_SIZE, sequence_size, EMBEDDING_DIM))\n","seq_lens = torch.randint(high=sequence_size, size=(1, BATCH_SIZE))\n","print (x.shape)\n","print (seq_lens.shape)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["torch.Size([64, 8, 100])\n","torch.Size([1, 64])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Ufxzi5G0DScl"},"source":["<div align=\"left\">\n","<img src=\"https://raw.githubusercontent.com/GokuMohandas/MadeWithML/main/images/basics/rnn/vanilla.png\" width=\"500\">\n","</div>\n","\n","RNN forward pass for a single time step $X_t$:\n","\n","$h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h)$\n","\n","*where*:\n","* $W_{hh}$ = hidden units weights| $\\in \\mathbb{R}^{HXH}$ ($H$ is the hidden dim)\n","* $h_{t-1}$ = previous timestep's hidden state $\\in \\mathbb{R}^{NXH}$\n","* $W_{xh}$ = input weights| $\\in \\mathbb{R}^{EXH}$\n","* $X_t$ = input at time step t | $\\in \\mathbb{R}^{NXE}$ ($N$ is the batch size, $E$ is the embedding dim)\n","* $b_h$ = hidden units bias $\\in \\mathbb{R}^{HX1}$\n","* $h_t$ = output from RNN for timestep $t$\n","\n","> At the first time step, the previous hidden state $h_{t-1}$ can either be a zero vector (unconditioned) or initialized (conditioned). If we are conditioning the RNN, the first hidden state $h_0$ can belong to a specific condition or we can concat the specific condition to the randomly initialized hidden vectors at each time step. More on this in the subsequent notebooks on RNNs."]},{"cell_type":"code","metadata":{"id":"K-Eb1HdRhnmu"},"source":["RNN_HIDDEN_DIM = 128\n","DROPOUT_P = 0.1\n","RNN_DROPOUT_P = 0.1"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"wJmu7XUTsc3F","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608336601047,"user_tz":420,"elapsed":19245,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"9302d1f7-5273-4526-ada6-8592134f8dc7"},"source":["# Initialize hidden state\n","hidden_t = torch.zeros((BATCH_SIZE, RNN_HIDDEN_DIM))\n","print (hidden_t.size())"],"execution_count":null,"outputs":[{"output_type":"stream","text":["torch.Size([64, 128])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"6ujvpD062Fi6"},"source":["We'll show how to create an RNN cell using PyTorch's [`RNNCell`](https://pytorch.org/docs/stable/generated/torch.nn.RNNCell.html#torch.nn.RNNCell) and the more abstracted [`RNN`](https://pytorch.org/docs/stable/generated/torch.nn.RNN.html#torch.nn.RNN)."]},{"cell_type":"code","metadata":{"id":"z-O2Du78ut8J","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608336601047,"user_tz":420,"elapsed":19230,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"313d1945-caf5-4026-85ad-1ea59df94dc0"},"source":["# Initialize RNN cell\n","rnn_cell = nn.RNNCell(EMBEDDING_DIM, RNN_HIDDEN_DIM)\n","print (rnn_cell)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["RNNCell(100, 128)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"FqOtTdDx3Nz7","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608336601048,"user_tz":420,"elapsed":19215,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"a976cb81-64fd-4cd9-fe74-5f896fb42f0c"},"source":["# Forward pass through RNN\n","x = x.permute(1, 0, 2) # RNN needs batch_size to be at dim 1\n","\n","# Loop through the inputs time steps\n","hiddens = []\n","for t in range(sequence_size):\n","    hidden_t = rnn_cell(x[t], hidden_t)\n","    hiddens.append(hidden_t)\n","hiddens = torch.stack(hiddens)\n","hiddens = hiddens.permute(1, 0, 2) # bring batch_size back to dim 0\n","print (hiddens.size())"],"execution_count":null,"outputs":[{"output_type":"stream","text":["torch.Size([64, 8, 128])\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"V1rb4lsltO-c","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608336601048,"user_tz":420,"elapsed":19199,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"edee46f8-ca79-4e8e-9043-b1bc727a10eb"},"source":["# We also could've used a more abstracted layer\n","x = torch.rand((BATCH_SIZE, sequence_size, EMBEDDING_DIM))\n","rnn = nn.RNN(EMBEDDING_DIM, RNN_HIDDEN_DIM, batch_first=True)\n","out, h_n = rnn(x) # h_n is the last hidden state\n","print (\"out: \", out.shape)\n","print (\"h_n: \", h_n.shape)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["out:  torch.Size([64, 8, 128])\n","h_n:  torch.Size([1, 64, 128])\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"EYoA2Wbet9wr","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608336601272,"user_tz":420,"elapsed":19407,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"cfc03864-7687-49dd-e508-d0a60706f64b"},"source":["# The same tensors\n","print (out[:,-1,:])\n","print (h_n.squeeze(0))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["tensor([[-0.5056,  0.6157,  0.4275,  ...,  0.1804, -0.1480, -0.4822],\n","        [-0.1490,  0.6549,  0.3184,  ...,  0.2831, -0.3557, -0.5438],\n","        [-0.5290,  0.4321,  0.0885,  ...,  0.4848, -0.2672, -0.2660],\n","        ...,\n","        [-0.2987,  0.4640, -0.1172,  ...,  0.5371, -0.3099, -0.4606],\n","        [-0.1882,  0.4102,  0.0346,  ...,  0.4146, -0.0700, -0.2791],\n","        [-0.4272,  0.2846, -0.0693,  ...,  0.3417, -0.1442, -0.3587]],\n","       grad_fn=<SliceBackward>)\n","tensor([[-0.5056,  0.6157,  0.4275,  ...,  0.1804, -0.1480, -0.4822],\n","        [-0.1490,  0.6549,  0.3184,  ...,  0.2831, -0.3557, -0.5438],\n","        [-0.5290,  0.4321,  0.0885,  ...,  0.4848, -0.2672, -0.2660],\n","        ...,\n","        [-0.2987,  0.4640, -0.1172,  ...,  0.5371, -0.3099, -0.4606],\n","        [-0.1882,  0.4102,  0.0346,  ...,  0.4146, -0.0700, -0.2791],\n","        [-0.4272,  0.2846, -0.0693,  ...,  0.3417, -0.1442, -0.3587]],\n","       grad_fn=<SqueezeBackward1>)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"3WJ_ZqzAu6Uk"},"source":["In our model, we want to use the RNN's output after the last relevant token in the sentence is processed. The last relevant token doesn't refer the `<PAD>` tokens but to the last actual word in the sentence and its index is different for each input in the batch. This is why we included a `seq_lens` tensor in our batches."]},{"cell_type":"code","metadata":{"id":"PqPOY6qbu6e0"},"source":["def gather_last_relevant_hidden(hiddens, seq_lens):\n","    \"\"\"Extract and collect the last relevant \n","    hidden state based on the sequence length.\"\"\"\n","    seq_lens = seq_lens.long().detach().cpu().numpy() - 1\n","    out = []\n","    for batch_index, column_index in enumerate(seq_lens):\n","        out.append(hiddens[batch_index, column_index])\n","    return torch.stack(out)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"MDIwDHJOu7Kj","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608336601273,"user_tz":420,"elapsed":19383,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"f055a2d6-5e4a-4830-83e1-d1ff1be699a6"},"source":["# Get the last relevant hidden state\n","gather_last_relevant_hidden(hiddens=out, seq_lens=seq_lens).squeeze(0).shape"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["torch.Size([64, 128])"]},"metadata":{"tags":[]},"execution_count":44}]},{"cell_type":"markdown","metadata":{"id":"mnUh3iRpJ2j8"},"source":["There are many different ways to use RNNs. So far we've processed our inputs one timestep at a time and we could either use the RNN's output at each time step or just use the final input timestep's RNN output. Let's look at a few other possibilities.\n","\n","<div align=\"left\">\n","<img src=\"https://raw.githubusercontent.com/GokuMohandas/MadeWithML/main/images/basics/rnn/architectures.png\" width=\"1000\">\n","</div>"]},{"cell_type":"markdown","metadata":{"id":"pfhjWZRD94hK"},"source":["## Model"]},{"cell_type":"code","metadata":{"id":"OXDA4UNdxmZT"},"source":["import torch.nn.functional as F"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"qoBC0u8XpbVv"},"source":["HIDDEN_DIM = 100"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"UPP5ROd69mXC"},"source":["class RNN(nn.Module):\n","    def __init__(self, embedding_dim, vocab_size, rnn_hidden_dim,\n","                 hidden_dim, dropout_p, num_classes, padding_idx=0):\n","        super(RNN, self).__init__()\n","        \n","        # Initialize embeddings\n","        self.embeddings = nn.Embedding(\n","            embedding_dim=embedding_dim, num_embeddings=vocab_size,\n","            padding_idx=padding_idx)\n","        \n","        # RNN\n","        self.rnn = nn.RNN(embedding_dim, rnn_hidden_dim, batch_first=True)\n","     \n","        # FC weights\n","        self.dropout = nn.Dropout(dropout_p)\n","        self.fc1 = nn.Linear(rnn_hidden_dim, hidden_dim)\n","        self.fc2 = nn.Linear(hidden_dim, num_classes)\n","\n","    def forward(self, inputs, apply_softmax=False):\n","        # Embed\n","        x_in, seq_lens = inputs\n","        x_in = self.embeddings(x_in)\n","            \n","        # Rnn outputs\n","        out, h_n = self.rnn(x_in)\n","        z = gather_last_relevant_hidden(hiddens=out, seq_lens=seq_lens)\n","\n","        # FC layers\n","        z = self.fc1(z)\n","        z = self.dropout(z)\n","        y_pred = self.fc2(z)\n","        \n","        if apply_softmax:\n","            y_pred = F.softmax(y_pred, dim=1)\n","        return y_pred"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"LH_z2FqUn5DR","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608336601275,"user_tz":420,"elapsed":19344,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"b1f97327-4295-4117-c298-a87a6d23f0cd"},"source":["# Simple RNN cell\n","model = RNN(\n","    embedding_dim=EMBEDDING_DIM, vocab_size=VOCAB_SIZE, \n","    rnn_hidden_dim=RNN_HIDDEN_DIM, hidden_dim=HIDDEN_DIM, \n","    dropout_p=DROPOUT_P, num_classes=NUM_CLASSES)\n","model = model.to(device) # set device\n","print (model.named_parameters)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["<bound method Module.named_parameters of RNN(\n","  (embeddings): Embedding(5000, 100, padding_idx=0)\n","  (rnn): RNN(100, 128, batch_first=True)\n","  (dropout): Dropout(p=0.1, inplace=False)\n","  (fc1): Linear(in_features=128, out_features=100, bias=True)\n","  (fc2): Linear(in_features=100, out_features=4, bias=True)\n",")>\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"S9ggYO6yHIm2"},"source":["## Training"]},{"cell_type":"code","metadata":{"id":"geKOPVzVK6S9"},"source":["from torch.optim import Adam"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"OfSBvenCiIce"},"source":["NUM_LAYERS = 1\n","LEARNING_RATE = 1e-4\n","PATIENCE = 10\n","NUM_EPOCHS = 50"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"4ilmlIMUzNIj"},"source":["# Define Loss\n","class_weights_tensor = torch.Tensor(list(class_weights.values())).to(device)\n","loss = nn.CrossEntropyLoss(weight=class_weights_tensor)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"Ucn3tYq1_sE1"},"source":["# Define optimizer & scheduler\n","optimizer = Adam(model.parameters(), lr=LEARNING_RATE) \n","scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n","    optimizer, mode='min', factor=0.1, patience=3)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"X5ac-uJXb-F7"},"source":["# Trainer module\n","trainer = Trainer(\n","    model=model, device=device, loss_fn=loss_fn, \n","    optimizer=optimizer, scheduler=scheduler)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"F-uKcZrABKVZ","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608337013563,"user_tz":420,"elapsed":431580,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"e1ac43f2-559f-4c06-f736-b95d7ece9f6d"},"source":["# Train\n","best_model = trainer.train(\n","    NUM_EPOCHS, PATIENCE, train_dataloader, val_dataloader)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Epoch: 1 | train_loss: 1.25605, val_loss: 1.10880, lr: 1.00E-04, _patience: 10\n","Epoch: 2 | train_loss: 1.03074, val_loss: 0.96749, lr: 1.00E-04, _patience: 10\n","Epoch: 3 | train_loss: 0.90110, val_loss: 0.86424, lr: 1.00E-04, _patience: 10\n","Epoch: 4 | train_loss: 0.80128, val_loss: 0.78586, lr: 1.00E-04, _patience: 10\n","Epoch: 5 | train_loss: 0.72344, val_loss: 0.72544, lr: 1.00E-04, _patience: 10\n","Epoch: 6 | train_loss: 0.66466, val_loss: 0.68083, lr: 1.00E-04, _patience: 10\n","Epoch: 7 | train_loss: 0.61909, val_loss: 0.64895, lr: 1.00E-04, _patience: 10\n","Epoch: 8 | train_loss: 0.58306, val_loss: 0.62504, lr: 1.00E-04, _patience: 10\n","Epoch: 9 | train_loss: 0.55312, val_loss: 0.60610, lr: 1.00E-04, _patience: 10\n","Epoch: 10 | train_loss: 0.52836, val_loss: 0.59164, lr: 1.00E-04, _patience: 10\n","Epoch: 11 | train_loss: 0.50698, val_loss: 0.58117, lr: 1.00E-04, _patience: 10\n","Epoch: 12 | train_loss: 0.48741, val_loss: 0.57346, lr: 1.00E-04, _patience: 10\n","Epoch: 13 | train_loss: 0.47061, val_loss: 0.56581, lr: 1.00E-04, _patience: 10\n","Epoch: 14 | train_loss: 0.45512, val_loss: 0.56082, lr: 1.00E-04, _patience: 10\n","Epoch: 15 | train_loss: 0.44111, val_loss: 0.55557, lr: 1.00E-04, _patience: 10\n","Epoch: 16 | train_loss: 0.42847, val_loss: 0.55547, lr: 1.00E-04, _patience: 10\n","Epoch: 17 | train_loss: 0.41675, val_loss: 0.55268, lr: 1.00E-04, _patience: 10\n","Epoch: 18 | train_loss: 0.40476, val_loss: 0.55213, lr: 1.00E-04, _patience: 10\n","Epoch: 19 | train_loss: 0.39431, val_loss: 0.55190, lr: 1.00E-04, _patience: 10\n","Epoch: 20 | train_loss: 0.38407, val_loss: 0.55570, lr: 1.00E-04, _patience: 9\n","Epoch: 21 | train_loss: 0.37476, val_loss: 0.55518, lr: 1.00E-04, _patience: 8\n","Epoch: 22 | train_loss: 0.36541, val_loss: 0.55668, lr: 1.00E-04, _patience: 7\n","Epoch: 23 | train_loss: 0.35682, val_loss: 0.56100, lr: 1.00E-05, _patience: 6\n","Epoch: 24 | train_loss: 0.33652, val_loss: 0.53492, lr: 1.00E-05, _patience: 10\n","Epoch: 25 | train_loss: 0.33039, val_loss: 0.53516, lr: 1.00E-05, _patience: 9\n","Epoch: 26 | train_loss: 0.32905, val_loss: 0.53654, lr: 1.00E-05, _patience: 8\n","Epoch: 27 | train_loss: 0.32716, val_loss: 0.53769, lr: 1.00E-05, _patience: 7\n","Epoch: 28 | train_loss: 0.32658, val_loss: 0.53878, lr: 1.00E-06, _patience: 6\n","Epoch: 29 | train_loss: 0.32324, val_loss: 0.53585, lr: 1.00E-06, _patience: 5\n","Epoch: 30 | train_loss: 0.32273, val_loss: 0.53579, lr: 1.00E-06, _patience: 4\n","Epoch: 31 | train_loss: 0.32206, val_loss: 0.53581, lr: 1.00E-06, _patience: 3\n","Epoch: 32 | train_loss: 0.32233, val_loss: 0.53587, lr: 1.00E-07, _patience: 2\n","Epoch: 33 | train_loss: 0.32215, val_loss: 0.53572, lr: 1.00E-07, _patience: 1\n","Stopping early!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"axytMjGQmz6T"},"source":["## Evaluation"]},{"cell_type":"code","metadata":{"id":"-Ly3zjVAm4Wr"},"source":["import json\n","from sklearn.metrics import precision_recall_fscore_support"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"ojOfz9rjm4ZI"},"source":["def get_performance(y_true, y_pred, classes):\n","    \"\"\"Per-class performance metrics.\"\"\"\n","    # Performance\n","    performance = {\"overall\": {}, \"class\": {}}\n","\n","    # Overall performance\n","    metrics = precision_recall_fscore_support(y_true, y_pred, average=\"weighted\")\n","    performance[\"overall\"][\"precision\"] = metrics[0]\n","    performance[\"overall\"][\"recall\"] = metrics[1]\n","    performance[\"overall\"][\"f1\"] = metrics[2]\n","    performance[\"overall\"][\"num_samples\"] = np.float64(len(y_true))\n","\n","    # Per-class performance\n","    metrics = precision_recall_fscore_support(y_true, y_pred, average=None)\n","    for i in range(len(classes)):\n","        performance[\"class\"][classes[i]] = {\n","            \"precision\": metrics[0][i],\n","            \"recall\": metrics[1][i],\n","            \"f1\": metrics[2][i],\n","            \"num_samples\": np.float64(metrics[3][i]),\n","        }\n","\n","    return performance"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"tho3e_yQzqZ1"},"source":["# Get predictions\n","test_loss, y_true, y_prob = trainer.eval_step(dataloader=test_dataloader)\n","y_pred = np.argmax(y_prob, axis=1)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"fylHduxTK-0N","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608337013866,"user_tz":420,"elapsed":431847,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"a6ccd922-a403-4827-aaac-a278b270d4a5"},"source":["# Determine performance\n","performance = get_performance(\n","    y_true=y_test, y_pred=y_pred, classes=label_encoder.classes)\n","print (json.dumps(performance['overall'], indent=2))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["{\n","  \"precision\": 0.8171357577653572,\n","  \"recall\": 0.8176111111111112,\n","  \"f1\": 0.8171696173843819,\n","  \"num_samples\": 18000.0\n","}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"jR3trOIQcWY2"},"source":["# Gated RNNs: LSTMs & GRUs"]},{"cell_type":"markdown","metadata":{"id":"IqFsufNVch4W"},"source":["While our simple RNNs so far are great for sequentially processing our inputs, they have quite a few disadvantages. They commonly suffer from exploding or vanishing gradients as a result using the same set of weights ($W_{xh}$ and $W_{hh}$) with each timestep's input. During backpropagation, this can cause gradients to explode (>1) or vanish (<1). If you multiply any number greater than 1 with itself over and over, it moves towards infinity (exploding gradients) and similarly,  If you multiply any number less than 1 with itself over and over, it moves towards zero (vanishing gradients). To mitigate this issue, gated RNNs were devised to selectively retrain information. If you're interested in learning more of the specifics, this [post](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) is a must-read.\n","\n","There are two popular types of gated RNNs: Long Short-term Memory (LSTMs) units and Gated Recurrent Units (GRUs).\n","\n","<div align=\"left\">\n","<img src=\"https://raw.githubusercontent.com/GokuMohandas/MadeWithML/main/images/basics/rnn/gated.png\" width=\"600\">\n","</div>\n","<a href=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/\">Understanding LSTM Networks</a> - Chris Olah\n","\n","\n","> When deciding between LSTMs and GRUs, empirical performance is the best factor but in genreal GRUs offer similar performance with less complexity (less weights). \n","\n"]},{"cell_type":"code","metadata":{"id":"HXuW93ef7xcg","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608337013867,"user_tz":420,"elapsed":431835,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"300c987f-0a54-410b-d8c4-16604b67f4b3"},"source":["# Input\n","sequence_size = 8 # words per input\n","x = torch.rand((BATCH_SIZE, sequence_size, EMBEDDING_DIM))\n","print (x.shape)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["torch.Size([64, 8, 100])\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"PqVCqWv2fz4N"},"source":["# GRU \n","gru = nn.GRU(input_size=EMBEDDING_DIM, hidden_size=RNN_HIDDEN_DIM, batch_first=True)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"8-Va8pRB9Xn4","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608337013868,"user_tz":420,"elapsed":431816,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"4cf1ed15-3fb0-49d9-eec3-813cb17b1add"},"source":["# Forward pass\n","out, h_n = gru(x)\n","print (f\"out: {out.shape}\")\n","print (f\"h_n: {h_n.shape}\")"],"execution_count":null,"outputs":[{"output_type":"stream","text":["out: torch.Size([64, 8, 128])\n","h_n: torch.Size([1, 64, 128])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"-N9Ga-Jx874z"},"source":["## Bidirectional RNN"]},{"cell_type":"markdown","metadata":{"id":"v3HkL93688AB"},"source":["We can also have RNNs that process inputs from both directions (first token to last token and vice versa) and combine their outputs. This architecture is known as a bidirectional RNN. "]},{"cell_type":"code","metadata":{"id":"ne711WL888Qf"},"source":["# GRU \n","gru = nn.GRU(input_size=EMBEDDING_DIM, hidden_size=RNN_HIDDEN_DIM, \n","             batch_first=True, bidirectional=True)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"52sSVNd_9TwH","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608337014312,"user_tz":420,"elapsed":432241,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"b8816e79-b314-4139-b6b6-ac3509a2fba9"},"source":["# Forward pass\n","out, h_n = gru(x)\n","print (f\"out: {out.shape}\")\n","print (f\"h_n: {h_n.shape}\")"],"execution_count":null,"outputs":[{"output_type":"stream","text":["out: torch.Size([64, 8, 256])\n","h_n: torch.Size([2, 64, 128])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"lO7l6idz9wMx"},"source":["Notice that the output for each sample at each timestamp has size 256 (double the `RNN_HIDDEN_DIM`). This is because this includes both the forward and backward directions from the BiRNN. "]},{"cell_type":"markdown","metadata":{"id":"mZUXGLRs981z"},"source":["## Model"]},{"cell_type":"code","metadata":{"id":"8I-AEQmG-Ge-"},"source":["class GRU(nn.Module):\n","    def __init__(self, embedding_dim, vocab_size, rnn_hidden_dim,\n","                 hidden_dim, dropout_p, num_classes, padding_idx=0):\n","        super(GRU, self).__init__()\n","        \n","        # Initialize embeddings\n","        self.embeddings = nn.Embedding(embedding_dim=embedding_dim,\n","                                       num_embeddings=vocab_size,\n","                                       padding_idx=padding_idx)\n","        \n","        # RNN\n","        self.rnn = nn.GRU(embedding_dim, rnn_hidden_dim, \n","                          batch_first=True, bidirectional=True)\n","     \n","        # FC weights\n","        self.dropout = nn.Dropout(dropout_p)\n","        self.fc1 = nn.Linear(rnn_hidden_dim*2, hidden_dim)\n","        self.fc2 = nn.Linear(hidden_dim, num_classes)\n","\n","    def forward(self, inputs, apply_softmax=False):\n","        # Embed\n","        x_in, seq_lens = inputs\n","        x_in = self.embeddings(x_in)\n","            \n","        # Rnn outputs\n","        out, h_n = self.rnn(x_in)\n","        z = gather_last_relevant_hidden(hiddens=out, seq_lens=seq_lens)\n","\n","        # FC layers\n","        z = self.fc1(z)\n","        z = self.dropout(z)\n","        y_pred = self.fc2(z)\n","        \n","        if apply_softmax:\n","            y_pred = F.softmax(y_pred, dim=1)\n","        return y_pred"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"1-tSbP86-e_s","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608337014313,"user_tz":420,"elapsed":432222,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"d29de891-ccb7-4a3d-d08e-f2e6e2a7ef63"},"source":["# Simple RNN cell\n","model = GRU(\n","    embedding_dim=EMBEDDING_DIM, vocab_size=VOCAB_SIZE, \n","    rnn_hidden_dim=RNN_HIDDEN_DIM, hidden_dim=HIDDEN_DIM, \n","    dropout_p=DROPOUT_P, num_classes=NUM_CLASSES)\n","model = model.to(device) # set device\n","print (model.named_parameters)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["<bound method Module.named_parameters of GRU(\n","  (embeddings): Embedding(5000, 100, padding_idx=0)\n","  (rnn): GRU(100, 128, batch_first=True, bidirectional=True)\n","  (dropout): Dropout(p=0.1, inplace=False)\n","  (fc1): Linear(in_features=256, out_features=100, bias=True)\n","  (fc2): Linear(in_features=100, out_features=4, bias=True)\n",")>\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"biToCtZw-oPq"},"source":["## Training"]},{"cell_type":"code","metadata":{"id":"LByz0XL1-pT1"},"source":["# Define Loss\n","class_weights_tensor = torch.Tensor(list(class_weights.values())).to(device)\n","loss = nn.CrossEntropyLoss(weight=class_weights_tensor)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"PMUYuLaI-shJ"},"source":["# Define optimizer & scheduler\n","optimizer = Adam(model.parameters(), lr=LEARNING_RATE) \n","scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n","    optimizer, mode='min', factor=0.1, patience=3)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"G3jTgXy4-wDw"},"source":["# Trainer module\n","trainer = Trainer(\n","    model=model, device=device, loss_fn=loss_fn, \n","    optimizer=optimizer, scheduler=scheduler)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"pH_qGGHq-0K5","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608337378205,"user_tz":420,"elapsed":796078,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"9ccfee80-cd96-4703-db7c-359edc66a2b6"},"source":["# Train\n","best_model = trainer.train(\n","    NUM_EPOCHS, PATIENCE, train_dataloader, val_dataloader)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Epoch: 1 | train_loss: 1.18125, val_loss: 0.93827, lr: 1.00E-04, _patience: 10\n","Epoch: 2 | train_loss: 0.81291, val_loss: 0.72564, lr: 1.00E-04, _patience: 10\n","Epoch: 3 | train_loss: 0.65413, val_loss: 0.64487, lr: 1.00E-04, _patience: 10\n","Epoch: 4 | train_loss: 0.57867, val_loss: 0.60529, lr: 1.00E-04, _patience: 10\n","Epoch: 5 | train_loss: 0.53073, val_loss: 0.57995, lr: 1.00E-04, _patience: 10\n","Epoch: 6 | train_loss: 0.49491, val_loss: 0.56374, lr: 1.00E-04, _patience: 10\n","Epoch: 7 | train_loss: 0.46601, val_loss: 0.55225, lr: 1.00E-04, _patience: 10\n","Epoch: 8 | train_loss: 0.44207, val_loss: 0.54532, lr: 1.00E-04, _patience: 10\n","Epoch: 9 | train_loss: 0.42205, val_loss: 0.54102, lr: 1.00E-04, _patience: 10\n","Epoch: 10 | train_loss: 0.40335, val_loss: 0.53801, lr: 1.00E-04, _patience: 10\n","Epoch: 11 | train_loss: 0.38722, val_loss: 0.53863, lr: 1.00E-04, _patience: 9\n","Epoch: 12 | train_loss: 0.37248, val_loss: 0.53962, lr: 1.00E-04, _patience: 8\n","Epoch: 13 | train_loss: 0.35893, val_loss: 0.54258, lr: 1.00E-04, _patience: 7\n","Epoch: 14 | train_loss: 0.34589, val_loss: 0.54491, lr: 1.00E-05, _patience: 6\n","Epoch: 15 | train_loss: 0.31934, val_loss: 0.53648, lr: 1.00E-05, _patience: 10\n","Epoch: 16 | train_loss: 0.31508, val_loss: 0.53640, lr: 1.00E-05, _patience: 10\n","Epoch: 17 | train_loss: 0.31280, val_loss: 0.53745, lr: 1.00E-05, _patience: 9\n","Epoch: 18 | train_loss: 0.31139, val_loss: 0.53864, lr: 1.00E-05, _patience: 8\n","Epoch: 19 | train_loss: 0.30909, val_loss: 0.53980, lr: 1.00E-05, _patience: 7\n","Epoch: 20 | train_loss: 0.30777, val_loss: 0.54081, lr: 1.00E-06, _patience: 6\n","Epoch: 21 | train_loss: 0.30386, val_loss: 0.53895, lr: 1.00E-06, _patience: 5\n","Epoch: 22 | train_loss: 0.30399, val_loss: 0.53897, lr: 1.00E-06, _patience: 4\n","Epoch: 23 | train_loss: 0.30351, val_loss: 0.53904, lr: 1.00E-06, _patience: 3\n","Epoch: 24 | train_loss: 0.30332, val_loss: 0.53912, lr: 1.00E-07, _patience: 2\n","Epoch: 25 | train_loss: 0.30300, val_loss: 0.53909, lr: 1.00E-07, _patience: 1\n","Stopping early!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"vskwiiI3V3S6"},"source":["## Evaluation"]},{"cell_type":"code","metadata":{"id":"4OQ5pEVco3xb"},"source":["from pathlib import Path"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"Itq7lT9qV9Y8"},"source":["# Get predictions\n","test_loss, y_true, y_prob = trainer.eval_step(dataloader=test_dataloader)\n","y_pred = np.argmax(y_prob, axis=1)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"VT9C8rUyWAWF","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1608337378987,"user_tz":420,"elapsed":796829,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"21129d0c-0efe-4d3a-fb62-6ed9a32a25d2"},"source":["# Determine performance\n","performance = get_performance(\n","    y_true=y_test, y_pred=y_pred, classes=label_encoder.classes)\n","print (json.dumps(performance['overall'], indent=2))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["{\n","  \"precision\": 0.8192635071011053,\n","  \"recall\": 0.8196111111111111,\n","  \"f1\": 0.8192710197821547,\n","  \"num_samples\": 18000.0\n","}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"mbxUAecEo-QC"},"source":["# Save artifacts\n","dir = Path(\"gru\")\n","dir.mkdir(parents=True, exist_ok=True)\n","label_encoder.save(fp=Path(dir, 'label_encoder.json'))\n","tokenizer.save(fp=Path(dir, 'tokenizer.json'))\n","torch.save(best_model.state_dict(), Path(dir, 'model.pt'))\n","with open(Path(dir, 'performance.json'), \"w\") as fp:\n","    json.dump(performance, indent=2, sort_keys=False, fp=fp)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"yeiD1T_QZpdk"},"source":["## Inference"]},{"cell_type":"code","metadata":{"id":"OAyVsyDepD5-"},"source":["def get_probability_distribution(y_prob, classes):\n","    \"\"\"Create a dict of class probabilities from an array.\"\"\"\n","    results = {}\n","    for i, class_ in enumerate(classes):\n","        results[class_] = np.float64(y_prob[i])\n","    sorted_results = {k: v for k, v in sorted(\n","        results.items(), key=lambda item: item[1], reverse=True)}\n","    return sorted_results"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"gNxo3XPSpLgG","executionInfo":{"status":"ok","timestamp":1608337378989,"user_tz":420,"elapsed":796803,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"b020623d-2a12-496b-93a4-232ef756c061"},"source":["# Load artifacts\n","device = torch.device(\"cpu\")\n","label_encoder = LabelEncoder.load(fp=Path(dir, 'label_encoder.json'))\n","tokenizer = Tokenizer.load(fp=Path(dir, 'tokenizer.json'))\n","model = GRU(\n","    embedding_dim=EMBEDDING_DIM, vocab_size=VOCAB_SIZE, \n","    rnn_hidden_dim=RNN_HIDDEN_DIM, hidden_dim=HIDDEN_DIM, \n","    dropout_p=DROPOUT_P, num_classes=NUM_CLASSES)\n","model.load_state_dict(torch.load(Path(dir, 'model.pt'), map_location=device))\n","model.to(device)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["GRU(\n","  (embeddings): Embedding(5000, 100, padding_idx=0)\n","  (rnn): GRU(100, 128, batch_first=True, bidirectional=True)\n","  (dropout): Dropout(p=0.1, inplace=False)\n","  (fc1): Linear(in_features=256, out_features=100, bias=True)\n","  (fc2): Linear(in_features=100, out_features=4, bias=True)\n",")"]},"metadata":{"tags":[]},"execution_count":75}]},{"cell_type":"code","metadata":{"id":"50gWn5a5pQDK"},"source":["# Initialize trainer\n","trainer = Trainer(model=model, device=device)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"CvHtQ9S8pRQX","executionInfo":{"status":"ok","timestamp":1608337378991,"user_tz":420,"elapsed":796785,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"d48d9dce-3712-49f8-949e-5c3c5fdbdcde"},"source":["# Dataloader\n","text = \"The final tennis tournament starts next week.\"\n","X = tokenizer.texts_to_sequences([preprocess(text)])\n","print (tokenizer.sequences_to_texts(X))\n","y_filler = label_encoder.encode([label_encoder.classes[0]]*len(X))\n","dataset = Dataset(X=X, y=y_filler, max_filter_size=max_filter_size)\n","dataloader = dataset.create_dataloader(batch_size=batch_size)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["['final tennis tournament starts next week']\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"DM03X5w9pRTM","executionInfo":{"status":"ok","timestamp":1608337378992,"user_tz":420,"elapsed":796775,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"b964deba-b82d-475f-97a0-a96d6379ac60"},"source":["# Inference\n","y_prob = trainer.predict_step(dataloader)\n","y_pred = np.argmax(y_prob, axis=1)\n","label_encoder.decode(y_pred)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['Sports']"]},"metadata":{"tags":[]},"execution_count":78}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"5omEa3BHpRVZ","executionInfo":{"status":"ok","timestamp":1608337378993,"user_tz":420,"elapsed":796764,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"outputId":"d0e4deea-f1d5-4d0a-b65d-6e826cfcc892"},"source":["# Class distributions\n","prob_dist = get_probability_distribution(y_prob=y_prob[0], classes=label_encoder.classes)\n","print (json.dumps(prob_dist, indent=2))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["{\n","  \"Sports\": 0.49753469228744507,\n","  \"World\": 0.2925860285758972,\n","  \"Business\": 0.1932886838912964,\n","  \"Sci/Tech\": 0.01659061387181282\n","}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"4xknbwa-bW3U"},"source":["> We will learn how to get a little bit of interpretabiltiy with RNNs in the next lesson on attentional interfaces."]}]}